Machine-learning models to facilitate user retention for software applications

ABSTRACT

An ordered combination of machine-learning models may be used to identify users who are likely to abandon use of an application, predict the reasons why those users are likely to abandon, and identify intervening actions that the application can perform to reduce the probability that the users will abandon the application. For example, a first machine-learning model determines a retention-prediction value indicating a probability that the user will complete a target action in the application before a session terminates. If the retention-prediction value satisfies a threshold condition, a second machine-learning model determines a reason why the session is likely to terminate before the user completes the target action. A third machine-learning model determines an intervention action for the application to perform to increase the probability that the user will complete the target action before the session terminates.

BACKGROUND Field

Embodiments generally relate to systems for facilitating user retentionfor software applications, e.g., preventing users from abandoning anapplication. Specifically, embodiments disclosed herein describe asynergistic machine-learning system for predicting behavior ofapplication users and selecting intervention actions for the applicationto perform to facilitate user retention.

Description of the Related Art

With machine-learning models, computing systems can improve and refinefunctionality without explicitly being programmed. Given a set oftraining data, a machine-learning model can generate and refine afunction that determines a target attribute value based on one or moreinput features. For example, if a set of input features describes anautomobile and the target value is the automobile's gas mileage, amachine-learning model can be trained to predict gas mileage based onthe input features, such as the automobile's weight, tire size, numberof cylinders, and engine displacement.

Some machine-learning models are well suited for domains that involvenumerical features. Other machine-learning models, such as decisiontrees, lend themselves more readily to domains that involve categoricalfeatures.

The predictive accuracy a machine-learning model achieves ultimatelydepends on many factors. Ideally, training data for the machine-learningmodel should be representative of the population for which predictionsare desired (e.g., unbiased and correctly labeled). In addition,training data should include a large number of training instancesrelative to the number of features on which predictions are based andrelative to the range of possible values for each feature.

SUMMARY

One embodiment of the present disclosure includes a system comprising:one or more processors and memory storing one or more instructions that,when executed on the one or more processors, cause the system to: sendone or more pages for display to a user via a network during aninteraction session between the user and an application, wherein the oneor more web pages include elements for collecting response data from theuser; receive, via the pages, response data from the user, collect, viathe application, additional data that characterizes user behavior duringthe interaction session; generate a composite data set from the responsedata and the additional data; determine, via a first machine-learningmodel based on the composite data set, a retention-prediction valueindicating a probability that the user will complete a target action inthe application before the interaction session terminates; determinethat the retention-prediction value satisfies a threshold condition;determine, via a second machine-learning model based on the compositedata set, a reason why the interaction session is likely to terminatebefore the user completes the target action; determine, via a thirdmachine-learning model based on the composite data set, an interventionaction for increasing the probability that the user will complete thetarget action before the interaction session terminates; and perform,via the application, the intervention action.

Another embodiment provides a computer-readable storage medium havinginstructions, which, when executed on a processor, perform an operationthat generally includes: sending one or more pages for display to a uservia a network during an interaction session between the user and anapplication, wherein the one or more web pages include elements forcollecting response data from the user; receiving, via the pages, theresponse data from the user; collecting, via the application, additionaldata that characterizes user behavior during the interaction session;generating a composite data set from the response data and theadditional data; determining, via a first machine-learning model basedon the composite data set, a retention-prediction value indicating aprobability that the user will complete a target action in theapplication before the interaction session terminates; determining thatthe retention-prediction value satisfies a threshold condition;determining, via a second machine-learning model based on the compositedata set, a reason why the interaction session is likely to terminatebefore the user completes the target action; determining, via a thirdmachine-learning model based on the composite data set, an interventionaction for increasing the probability that the user will complete thetarget action before the interaction session terminates; and performing,via the application, the intervention action.

Another embodiment of the present disclosure includes a method thatgenerally includes: sending one or more pages for display to a user viaa network during an interaction session between the user and anapplication, wherein the one or more web pages include elements forcollecting response data from the user; receiving, via the pages, theresponse data from the user; collecting, via the application, additionaldata that characterizes user behavior during the interaction session;generating a composite data set from the response data and theadditional data; determining, via a first machine-learning model basedon the composite data set, a retention-prediction value indicating aprobability that the user will complete a target action in theapplication before the interaction session terminates; determining thatthe retention-prediction value satisfies a threshold condition;determining, via a second machine-learning model based on the compositedata set, a reason why the interaction session is likely to terminatebefore the user completes the target action; determining, via a thirdmachine-learning model based on the composite data set, an interventionaction for increasing the probability that the user will complete thetarget action before the interaction session terminates; and performing,via the application, the intervention action.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the manner in which the above recited features of the presentdisclosure can be understood in detail, a more particular description ofthe disclosure, briefly summarized above, may be had by reference toembodiments, some of which are illustrated in the appended drawings. Itis to be noted, however, that the appended drawings illustrate onlyexemplary embodiments and are therefore not to be considered limiting ofthe scope of the disclosure. The scope of the disclosure may admit toother embodiments.

FIG. 1 illustrates a computing network environment in which technologyof the present disclosure can operate, according to one embodiment.

FIG. 2 illustrates a detailed view of a retention module, according toone embodiment.

FIG. 3 illustrates a process for reducing user abandonment of anapplication, according to one embodiment.

FIG. 4 illustrates an application-user retention system 400, accordingto an embodiment.

To facilitate understanding, identical reference numerals have beenused, where possible, to designate identical elements that are common tothe figures. It is contemplated that elements and features of oneembodiment may be beneficially incorporated in other embodiments withoutfurther recitation.

DETAILED DESCRIPTION

Embodiments presented herein provide systems for reducing userapplication abandonment. Specifically, embodiments disclosed hereindescribe a machine-learning system for predicting behavior ofapplication users and selecting intervention actions for the applicationto perform to facilitate user retention.

Typically, software vendors and other entities that provide webapplications hope to achieve a desired outcome when users access thoseapplications. A desired outcome for a software vendor, for example, maybe that a user purchases some product or service offered through theapplication. In another example, a desired outcome for a researchorganization may be that a user completes and submits a set of surveyresponses through the application.

However, the desired outcome may not occur every time a user accessesthe application. If the user is confused by some aspect of theapplication, the user may decide to quit using the applicationprematurely. For example, many users abandon using tax-preparationsoftware before filing their tax returns because many users are confusedby the terminology used on tax forms presented within thetax-preparation software.

Systems described herein use an ordered combination machine-learningmodels to identify users who are likely to abandon use of anapplication, predict the reasons why those users are likely to abandon,and identify intervening actions that the application can perform toreduce the probability that the users will abandon the application.Furthermore, systems described herein may also predict the next actionthe user will perform within the application, pre-fetch data (e.g.,pages, instructions, etc.) related to the next action, and refrain fromfetching data unrelated to the next action prematurely. This canaccelerate application response time, promote efficient use of memoryand network bandwidth, and enhance the Quality of Experience (QoE) forthe user overall.

FIG. 1 illustrates a computing network environment 100 in whichtechnology of the present disclosure can operate, according to oneembodiment. As shown, the computing network environment 100 includes anetwork 102, a computing device 120, and a server 130. A web application132 is provided by the server 130. A browser 122 executes at thecomputing device 120 and communicates with the web application 132 viathe network 102. Functionality attributed to the web application 132 maybe applied using code executed by the server 130, code executed by thecomputing device 120 via the browser 122, or a combination of bothserver-side code and client-side code. Note that, in some embodiments, astandalone dedicated application that executes on the computing device120 may be used instead of the browser 122. Also, though the server 130is illustrated as a single computing device for clarity, the componentsand functions attributed to the server 130 may be spread across multipleservers and computing resources (e.g., in a cloud-computingenvironment).

The web application 132 may be configured to display a number of webpages in succession to a user via the browser 122 in order to collectresponse data from the user. For example, the web application 132 may bean Ajax-style application that uses objects defined in adata-interchange format such as Javascript Object Notation (JSON) todefine the pages. The objects may include logic and member functions,member data such as primitive attributes, objects, and other constructsthat define what will be displayed and what information will besolicited from the user on each page displayed.

After a user provides input through a first page of the web application132, successive pages may dynamically generate content displayed to theuser based on the input from the first page. The sequence of pages shownto a user may vary depending on the information the user provides. Forexample, if the web application 132 is tax-preparation application, auser who enters income recorded on a 1099-MISC form may be shown a pageasking the user to enter business expenses. However, the web application132 does not show this page for business expenses to a user who onlyenters income recorded on a W-2 form. By adjusting the sequence of pagesshown to the user in this fashion, the web application 132 ensures thatthe content displayed on the pages and the information gathered isrelevant to the user's tax return. Similarly, if the web application 132is software for determining a patient's medical history, a user whoindicates she has received an organ donation may be shown a follow-uppage asking for more details about the organ donation (e.g., the type oforgan transplanted, the date the transplant operation was performed, thetypes of immunosuppressant drugs prescribed, etc.). By contrast, a userwho indicates she has never received an organ transplant would not beshown the follow-up page.

Once a sufficient collection of response data has been collected fromthe user, the web application 132 can invite the user to complete atarget action. The nature of the target action may vary depending on thepurpose of the web application 132. For example, if the web application132 is software for preparing tax returns, the target action may be thatthe user purchases one or more tax-preparation services or productsoffered through the web application 132. However, in other examples, thetarget action may be something other than a purchase. For example, ifthe web application 132 is designed to conduct a survey, the targetaction may be for a user to click a button to submit survey responses.Hence, the subject matter described herein is not limited toimplementations that involve commerce. Embodiments in which the webapplication 132 is described as tax-preparation software are providedsolely as an example for illustrative purposes to facilitateunderstanding of the broader concepts disclosed herein.

In some cases, a user may discontinue using the web application 132(e.g., via the web browser 122) without completing the target action.For example, if the web application 132 is tax-preparation software, theuser may abandon using the web application 132 because the user isconfused by questions displayed on a page of the web application; or theuser may become dissatisfied with how long it is taking to finishentering the requested information; or the user may be unwilling to paya posted price for a product or service offered, which is necessary tocomplete the user's tax return. Regardless of the reason, if the userabandons using the web application 132, a potential or existing customermay be lost. Similarly, if the web application 132 is software forconducting surveys, the user may be confused or unwilling to respond toa question asked. In this example, potentially valuable response data islost if the user abandons using the web application 132.

To reduce or prevent user abandonment of the web application 132, theserver 130 includes a retention module 134. Throughout an interactionsession between the user and the application 132, the retention module134 determines a retention-prediction value at regular or varying timeintervals. The retention-prediction value may be the probability thatthe user will abandon using the application 132 before completing thetarget action. Alternatively, the retention-prediction value may be theprobability that the user will complete the target action (i.e., oneminus the probability that the user will abandon). In other embodiments,the retention-prediction value may be a numeric or categorical valuethat maps to either the probability of abandonment or the probability ofcompletion in a predefined scheme. Regardless, the retention-predictionvalue indicates the probability of abandonment or the probability ofcompletion in one way or another. In some embodiments, the retentionmodule 134 determines the retention-prediction value via a firstmachine-learning model (e.g., as described in greater detail in FIG. 2)that calculates the retention-prediction value based on input features.The input features may be based on (i.e., represented by, contained in,or derived from) a composite information set.

The composite information set may include both the explicit responsedata provided by the user and additional data that characterizes theuser's current interaction session with the web application 132. Forexample, in the context of an example tax preparation web application,the explicit response data may include values such as: a name, abirthdate, an address, a number of dependents, an income amount, one ormore tax deductions, one or more tax credits, a filing status, one ormore moving expenses, a federal tax withholding, or the like. Theadditional data may include, for example, the amount of time the userhas spent on a page currently being displayed; the amounts of time theuser spent on previously displayed pages, respectively; the cumulativeamount of time spent during the current interaction session; the numbersof different types of events (e.g., click events, mouse-over events,etc.) initiated by the user on each page; the cumulative numbers ofdifferent types of events initiated by the user during the interactionsession; how many times a “back” button in the browser 122 has beenclicked during the interaction session; how many times the user hasreturned to previously displayed pages; whether the user has changed orerased entries in input fields; how many times the user has clicked a“next” button to proceed to subsequent pages before entering data inrequired input fields on a current page; how many times a user hasclicked on a “help” icon; and other types of data related to the user'sinteraction with the web application 132. Furthermore, if the computingdevice 120 is a mobile device, the additional data may also include theuser's location (e.g., as indicated by a global positioning system (GPS)associated with the computing device 120), a rate of movement of thedevice based on GPS measurements (which may indicate that the user is ina hurry or attempting to multitask), and linguistic or paralinguisticfeatures of audio input received through a microphone associated withthe computing device 120 (which may indicate whether the user isfrustrated).

The retention module 134 updates the retention-prediction valuethroughout the interaction session as updated composite information(e.g., both response data and additional data) becomes available. Forexample, the retention module 134 may recalculate theretention-prediction value when the user provides input through anelement on a page (e.g., an input field or a button). In one embodiment,the retention module 134 recalculates the retention-prediction valueeach time a selected time interval passes. Also, each time the retentionmodule 134 recalculates the retention-prediction value, the retentionmodule 134 may also estimate a rate of change of theretention-prediction value. For example, the retention module 134 maysubtract a preceding retention-prediction value from the recalculatedretention-prediction value and divide the difference by the length ofthe time interval between the preceding value and the recalculated valueto estimate a rate at which the retention-prediction value is changing.If the rate of change suggests that the probability of the usercompleting the target action is decreasing at more than a thresholdrate, the retention module 134 may reduce the length of the timeinterval in proportion to the rate of change. As a result, the retentionmodule 134 may recalculate the retention-prediction value morefrequently if the probability of the user completing the target task isdecreasing faster than the threshold rate.

When the retention-prediction value is calculated or updated, theretention module 134 also compares the updated retention-predictionvalue to a threshold retention-prediction value. If the updatedretention-prediction value meets the threshold retention-predictionvalue (e.g., is less than or equal to the threshold retention-predictionvalue, is greater than or equal to the threshold retention-predictionvalue, etc.), the retention module 134 selects an intervention actionfor the web application 132 to perform. The purpose of the interventionaction is to dissuade the user from abandoning use of the webapplication 132, thereby increasing the probability that the user willperform the target action.

To select the intervention action, the retention module 134 firstdetermines, via a second machine-learning model (as described in greaterdetail in FIG. 2), a reason that the user is likely to fail to completethe target action. The second machine-learning model determines thereason based on the retention-prediction value and input features thatare based on the composite information set. The reason may be selectedfrom a predefined set of possible reasons (e.g., “confusion,” “cost toohigh,” “taking too much time,” “dissatisfied,” “language barrier,”etc.).

Once the reason has been determined, the retention module 134 determinesthe intervention action via a third machine-learning model (as describedin greater detail in FIG. 2) based on the retention-prediction value,the reason, and input features based on (e.g., derived from) thecomposite information set. The intervention action may be selected froma predefined set of possible intervention actions (e.g., “offerdiscount,” “connect with live assistance,” “change display language,”“open dialog box with additional explanation,” “show instructionaldemonstration video,” etc.).

Once the retention module 134 has determined the intervention action,the retention module 134 signals the web application 132 to perform theintervention action. For example, if the intervention action is toconnect the user with a live support agent, the web application 132 canopen a messaging interface (e.g., a chat window or a two-way audioconnection) via the browser 122. The messaging interface can allow theuser to communicate with a live support agent via the network 102. Inanother example, if the intervention action is to offer the user adiscount for a product or service, the web application 132 can presentthe discount offer in the browser 122 (e.g., in a sidebar, a pop-upwindow, or another page element).

At regular intervals during the interaction session, the retentionmodule 134 may also predict the next action a user will take andcustomize the page(s) and/or content presented to the user based on theprediction. The retention module 134 may predict the next action via afourth machine-learning model (as described in greater detail in FIG. 2)based on the retention-prediction value, the reason (if theretention-prediction value meets the threshold retention-predictionvalue), and input features based on the composite information set. Forexample, suppose the web application 132 is tax-preparation software andthe user enters a value in a text input field labeled “routing number”(e.g., to indicate that the user would like to receive a tax refund viadirect deposit). The retention module 134 may predict the user's nextaction will be to enter an account number in an adjacent text inputfield labeled “account number.” Based on this prediction, the webapplication 132 may customize the page to make the predicted actioneasier for the user to perform. For example, the web application 132 maydynamically increase the display size of the “account number” label andthe adjacent text input field, highlight the adjacent text field, ordisplay a pop-up balloon with instructions next to the adjacent textfield. By customizing the pages and content presented during theinteraction session in this manner, the web application 132 may providea more satisfactory quality of experience (QoE) for the user.

FIG. 2 illustrates a detailed view of the retention module 134,according to one embodiment. As shown, the retention module 134 includestraining data 210, goal-monitoring model 212, reason-determination model214, intervention-selection model 216, and anticipation model 218.Although the training data 210 is shown to be a component of theretention module 134 for simplicity, it should be understood that thetraining data 210 may be stored in a location that is remote from thelocation where software of the retention module 134 executes.

The training data 210 may comprise composite information, for example inthe form of composite data sets, describing previous interactionsessions between users and the web application 132. In addition, thetraining data 210 may also comprise attributes and labels that describeany intervention actions taken during those previous interactionsessions, whether the users completed the target actions for theprevious interactions sessions, and the underlying reasons (e.g., aspredicted by the third machine learning model 216 or as verified bysurveying the users) why any of those users did not complete the targetactions during the previous interaction sessions.

The goal-monitoring model 212 is a machine-learning model. Using thetraining data 210, the goal-monitoring model 212 is trained to predictretention-prediction values for interaction sessions using inputfeatures that are based on the composite data sets that describe thoseinteraction sessions. In one example, the composite data sets compriseactual input features. In another embodiment, input features aregenerated by applying one or more preprocessing steps (e.g.,normalizing, discretizing, quantifying, etc.) to information in thecomposite data sets. For example, categorical or ordinal data may beconverted to numeric data through known methods.

The retention-prediction value predicted for a particular interactionsession based on the associated input features indicates the probabilityof user abandonment without completion of the target action. Forexample, the retention-prediction may be a literal probability ofabandonment (e.g., a real number ranging from zero to one), aprobability of completion of the target action (e.g., a real numberequal to one minus the probability of abandonment), or some other typeof value (e.g., a categorical value or a discrete value) that can bemapped to a probability of abandonment according to a predefinedfunction available to the retention module 134.

A training instance may comprise a set (e.g., a vector) of inputfeatures and an associated target retention-prediction value. The targetretention-prediction value may be a label that has been empiricallyverified for the training instance. For example, if a user ultimatelyabandoned an interaction session described by a particular set of inputfeatures, a training instance including those input features can belabeled with a zero (the target retention-prediction value) to indicatethe target action was not completed or a one (or some other non-zerovalue) to indicate the target action was completed. The composite dataset for a single interaction session may be used to generate multipletraining instances. For example, an interaction session takes place overa definite period of time. The period of time may be defined by a firsttimestamp when the interaction session commenced and a second timestampwhen the interaction session terminated. A first training instance mayinclude input features that describe the interaction session for a firsttime window ranging from the first timestamp to a first point in timethat lies between the first timestamp and the second timestamp. A secondtraining instance may include input features that describe theinteraction session for a second time window ranging from the firsttimestamp to a second point in time that lies between the first point intime and the second timestamp. Subsequent training instances based onthe interaction session may be defined according to the same pattern,where each subsequent training instance includes input features thatdescribe the interaction session for a time window ranging from thefirst timestamp to a point in time that lies between the end of apreceding time window (of a preceding training instance) and the secondtimestamp. When arranged in chronological order, training instancesbased on the same interaction session make up a time series for trainingthe goal-monitoring model 212. Time-series training instancescorresponding to many different interaction sessions may be used totrain the goal-monitoring model 212.

In general, for most types of machine-learning models to achieve robustprediction accuracy, the number of input features in any particulartraining instance should be small relative to the total number oftraining instances used to train the machine-learning model. In someembodiments, the number of features for each training instance can be atleast three orders of magnitude smaller than the total number oftraining instances used to train the machine-learning model.

There are many different types of inductive and transductivemachine-learning models that can be used for the goal-monitoring model212, including: adsorption models, neural networks, support vectormachines, Bayesian belief networks, association-rule models, decisiontrees, nearest-neighbor models (e.g., k-NN), regression models,artificial neural networks, deep belief networks, and Q-learning models,among others.

Many configurations and parameter combinations may be possible for agiven type of machine-learning model. For example, with a neuralnetwork, the number of hidden layers, the number of hidden nodes in eachlayer, and the existence of recurrence relationships between layers canvary. Batch gradient descent or stochastic gradient descent may be usedin the process of tuning weights for the nodes in the neural network.The learning rate parameter, which partially determines how much eachweight may be adjusted at each step, may be varied. Input features maybe normalized. Other parameters that are known in the art, such asmomentum, may also be applied to improve neural network performance.

In another example, decision trees can be constructed using a variety ofapproaches, including: the iterative dichotomiser 3 (ID3),Classification and Regression Tree (CART), and CHi-squared AutomaticInteraction Detection (CHAID) methods. These methods may use one or moredifferent metrics to determine the order in which attribute values areexamined in decision trees, including: information gain and Giniimpurity. In addition, pruning methods may be applied to improvedecision tree performance, including: reduced error pruning, and costcomplexity pruning.

Furthermore, individual machine learning models can be combined to forman ensemble machine-learning model. An ensemble machine-learning modelmay be homogenous (i.e., using multiple member models of the same type)or non-homogenous (i.e., using multiple member models of differenttypes). Individual machine-learning models within an ensemble may all betrained using the same training data or may be trained using overlappingor non-overlapping subsets randomly selected from a larger set oftraining data. The Random-Forest model, for example, is an ensemblemodel in which multiple decision trees are generated using randomizedsubsets of input features and randomized subsets of training instances.

Since the training data 210 can be used to generate time-series traininginstances from the composite data sets that describe previousinteraction sessions, the goal-monitoring model 212 may be a deeprecurrent long short-term memory (LSTM) model. When instances in a timeseries are fed into an LSTM model in chronological order, the LSTM canrecall information (e.g., input features and predictedretention-prediction values) found in previous instances that have beeninput into the LSTM. When predicting a current retention-predictionvalue for a current instance, the LSTM can use both the input featuresof the current instance and the recalled information from previousinstances to predict the current retention-prediction value moreaccurately. Though a full explanation of LSTM models is beyond the scopeof this disclosure, LSTM models can be generated using publiclyavailable software libraries, such as TENSORFLOW™.

The reason-determination model 214 is also a machine-learning model. Thereason-determination model 214 is trained to predict a reason that auser is likely to fail to complete a target action for an interactionsession. Like the goal-monitoring model 212, the reason-determinationmodel 214 is trained using input features that are based on thecomposite data sets that describe interaction sessions. In one example,the composite data sets comprise actual input features used for thereason-determination model 214. In another embodiment, input featuresare generated by applying one or more preprocessing steps (e.g.,normalizing, discretizing, etc.) to information in the composite datasets. The input features for the reason-determination model 214 may alsoinclude the retention-prediction value predicted by the goal-monitoringmodel 212 for a given interaction session (or time window thereof).

A set (e.g., a vector) of input features and an associated target reason(i.e., a known reason indicating why a user abandoned) make up atraining instance for the reason-determination model 214. As explainedabove with respect to the goal-monitoring model 212, the composite dataset for a single interaction session may be used to generate multipletraining instances for the reason-determination model 214. For example,the composite data set for a single interaction session may be used togenerate multiple training instances that make up a time series. Likethe goal-monitoring model 212, the reason-determination model 214 may bean LSTM model or another type of machine-learning model.

The intervention-selection model 216 is also a machine-learning model.The intervention-selection model 216 is trained to predict anintervention action that will increase the probability that a user willcomplete a target action during an interaction session. Like thegoal-monitoring model 212, the intervention-selection model 216 istrained using input features that are based on the composite data setsof the training data 210 that describe interaction sessions. In oneexample, the composite data sets include actual input features used forthe intervention-selection model 216. In another example, input featuresare generated by applying one or more preprocessing steps (e.g.,normalizing, discretizing, etc.) to information in the composite datasets. The input features for the intervention-selection model 216 mayalso include the retention-prediction value predicted by thegoal-monitoring model 212 and the reason predicted by thereason-determination model 214 for a given interaction session (or timewindow thereof).

A set of input features and an associated target intervention action(i.e., an intervention action used during an interaction session inwhich a user completed a target action) make up a training instance forthe intervention-selection model 216. As explained above with respect tothe goal-monitoring model 212, the composite data set for a singleinteraction session may be used to generate multiple training instancesfor the intervention-selection model 216. The composite data set for asingle interaction session may be used to generate multiple traininginstances that make up a time series. Like the goal-monitoring model212, the intervention-selection model 216 may be an LSTM model oranother type of machine-learning model.

The anticipation model 218 is trained to predict the next action that auser will take during an interaction session. Like the goal-monitoringmodel 212, the anticipation model 218 is trained using input featuresthat are based on the composite data sets that describe interactionsessions (e.g., found in training data 210). In one example, thecomposite data sets contain actual input features used for theanticipation model 218. In another example, input features are generatedby applying one or more preprocessing steps (e.g., normalizing,discretizing, etc.) to information in the composite data sets. The inputfeatures for the anticipation model 218 may also include theretention-prediction value predicted by the goal-monitoring model 212,the reason predicted by the reason-determination model 214, or theintervention action selected by the intervention-selection model 216 fora given interaction session (or time window thereof).

A set of input features and an associated next action (i.e., the nextaction that the user actually performed in a next time window, asdescribed in the training data 210) make up a training instance for theanticipation model 218. As explained above with respect to thegoal-monitoring model 212, the composite data set for a singleinteraction session may be used to generate multiple training instancesfor the anticipation model 218. The composite data set for a singleinteraction session may be used to generate multiple training instancesthat make up a time series. Like the goal-monitoring model 212, theanticipation model 218 may be an LSTM model or another type ofmachine-learning model.

FIG. 3 is a flow diagram illustrating a process 300 for reducing userabandonment of an application, according to one embodiment.

As shown in block 302, the process 300 includes sending one or morepages for display to a user via a network during an interaction sessionbetween the user and an application. The application may be a webapplication executing on a server and the pages may be displayed to theuser on a client device in a browser or in a dedicated application. Theone or more pages include elements for collecting response data from theuser, such as input fields for text, buttons, check boxes, radiobuttons, and the like.

As shown in block 304, the process 300 includes receiving, via thepages, the response data from the user. The response data may includeinformation explicitly provided by the user (e.g., name, birthdate,address, preferences, etc.).

As shown in block 306, the process 300 includes collecting, via theapplication, additional data that characterizes user behavior during theinteraction session. The additional data may include, for example, theamount of time the user has spent on a page currently being displayed;the amounts of time the user spent on previously displayed pages,respectively; the cumulative amount of time spent during the currentinteraction session; the numbers of different types of events (e.g.,click events, mouse-over events, etc.) initiated by the user on eachpage; the cumulative numbers of different types of events initiated bythe user during the interaction session; how many times a “back” buttonin the browser has been clicked during the interaction session; how manytimes the user has returned to previously displayed pages; whether theuser has changed or erased entries in input fields; how many times theuser has clicked a “next” button to proceed to subsequent pages beforeentering data in required input fields on a current page; how many timesa user has clicked on a “help” icon; and other types of data.Furthermore, if the computing device is a mobile device, the additionaldata may also include the user's location (e.g., as indicated by aglobal positioning system (GPS) associated with the computing device), arate of movement of the device based on GPS measurements (which mayindicate that the user is in a hurry or attempting to multitask), andlinguistic or paralinguistic features of audio input received through amicrophone associated with the computing device (which may indicatewhether the user is frustrated).

As shown in block 308, the process 300 includes generating a compositedata set from the response data and the additional data.

As shown in block 310, the process 300 includes determining, via a firstmachine-learning model based on the composite data set, aretention-prediction value indicating a probability that the user willcomplete a target action in the application before the interactionsession terminates. If a previous retention-prediction value for theinteraction session has been calculated, the retention-prediction valuedetermined in block 310 may be an updated retention-prediction valuesbased on an updated composite data set. In such a case, the operations300 may also include subtracting the previous retention-prediction valuefrom the updated retention-prediction value to determine a difference;dividing the difference by a time interval to determine a rate ofchange; and determining an updated time interval based on the rate ofchange. Once the updated time interval elapses, another updatedretention-prediction can be determined.

As shown in block 312, the process 300 includes determining whether theretention-prediction value satisfies a threshold condition. Thethreshold condition may be, for example, that the retention-predictionvalue is less than or equal to a threshold retention-prediction value.If the threshold condition is satisfied, the flow of process 300proceeds to block 314. Otherwise, the flow of process proceeds back toblock 304.

As shown in block 314, the process 300 includes determining, via asecond machine-learning model based on the composite data set, a reasonwhy the interaction session is likely to terminate before the usercompletes the target action. The reason may be selected from apredefined set of possible reasons (e.g., a set comprising reasons suchas “confusion,” “cost too high,” “taking too much time,” “dissatisfied,”“language barrier,” etc.). Determining the reason via the secondmachine-learning model based on the composite data set may includeinputting a set of input features that includes the retention-predictionvalue into the second machine-learning model.

As shown in block 316, the process 300 includes determining, via a thirdmachine-learning model based on the composite data set, an interventionaction for increasing the probability that the user will complete thetarget action before the interaction session terminates. Theintervention action may be selected from a predefined set of possibleinterventions (e.g., a set comprising intervention actions such as“offer discount,” “connect with live assistance,” “change displaylanguage,” “open dialog box with additional explanation,” “showinstructional demonstration video,” etc.). Determining the interventionaction via the third machine-learning model may include inputting a setof input features that includes the retention-prediction value or thereason into the third machine-learning model.

As shown in block 318, the process 300 includes performing, via theapplication, the intervention action. Performing the intervention actionmay include opening, via the application, a messaging interface andestablishing a network connection with a live support agent to allow thelive support agent communicate with the user through the messaginginterface.

Optionally, the process 300 may also include determining, via a fourthmachine-learning model based on the composite data set, a next actionthe user is anticipated to perform in the application and altering atleast one aspect of the one or more web pages to facilitate userperformance of the next action. For example, if the next action is thatthe user will enter text in a field, the application may dynamicallyincrease the display size of a label of the field and/or the fielditself, highlight the field, or display a pop-up balloon withinstructions near the field to explain a format in which text should beentered into the field. If the next action is that the user will click abutton labeled “save changes,” the application can identify any elementson the page marked as “response required” for which the user has not yetprovided a response, highlight the elements, and display a messageexplaining that the user must provide the responses before moving on tothe next page. Also, the application can determine which page to displaynext based on the user's inputs.

Predicting the next action the user will perform allows the applicationto pre-fetch data (e.g., pages, instructions, etc.) related to the nextaction and to avoid fetching data unrelated to the next actionprematurely. This can accelerate application response time, promoteefficient use of memory and network bandwidth, and enhance the QoE forthe user overall.

FIG. 4 illustrates an application-user retention system 400, accordingto an embodiment. As shown, the application-user retention system 400includes a central processing unit (CPU) 402, at least one I/O deviceinterface 404 which may allow for the connection of various I/O devices414 (e.g., keyboards, displays, mouse devices, pen input, speakers,microphones, motion sensors, etc.) to the application-user retentionsystem 400, network interface 406, a memory 408, storage 410, and aninterconnect 412.

CPU 402 may retrieve and execute programming instructions stored in thememory 408. Similarly, the CPU 402 may retrieve and store applicationdata residing in the memory 408. The interconnect 412 transmitsprogramming instructions and application data, among the CPU 402, I/Odevice interface 404, network interface 406, memory 408, and storage410. CPU 402 can represent a single CPU, multiple CPUs, a single CPUhaving multiple processing cores, and the like. Additionally, the memory408 represents random access memory. Furthermore, the storage 410 may bea disk drive. Although shown as a single unit, the storage 410 may be acombination of fixed or removable storage devices, such as fixed discdrives, removable memory cards or optical storage, network attachedstorage (NAS), or a storage area-network (SAN).

As shown, memory 408 includes an application 416 and a retention module418. Storage 410 includes training data 420.

The application-user retention system 400 can operate in the followingmanner. The application 416 sends one or more pages for display to auser (e.g., via I/O device interface 404 or network interface 406) aspart of an interaction session between the user and the application 416.The one or more web pages include elements for collecting response datafrom the user. The application 416 receives the response data from theuser (e.g., via I/O device interface 404 or network interface 406) inresponse to the sending. The application 416 also collects additionaldata that characterizes user behavior during the interaction session.The application 416 generates a composite data set from the responsedata and the additional data.

Next, the retention module 418 determines, via a first machine-learningmodel based on the composite data set, a retention-prediction valueindicating a probability that the user will complete a target actionassociated with the application 416 before the interaction sessionterminates. If that the retention-prediction value satisfies a thresholdcondition, the retention module 418 determines, via a secondmachine-learning model based on the composite data set, a reason why theinteraction session is likely to terminate before the user completes thetarget action. Furthermore, the retention module 418 determines, via athird machine-learning model based on the composite data set, anintervention action for increasing the probability that the user willcomplete the target action before the interaction session terminates.The application 416 performs the intervention action.

Note, descriptions of embodiments of the present disclosure arepresented above for purposes of illustration, but embodiments of thepresent disclosure are not intended to be limited to any of thedisclosed embodiments. Many modifications and variations will beapparent to those of ordinary skill in the art without departing fromthe scope and spirit of the described embodiments. The terminology usedherein was chosen to best explain the principles of the embodiments, thepractical application or technical improvement over technologies foundin the marketplace, or to enable others of ordinary skill in the art tounderstand the embodiments disclosed herein.

In the preceding, reference is made to embodiments presented in thisdisclosure. However, the scope of the present disclosure is not limitedto specific described embodiments. Instead, any combination of thefollowing features and elements, whether related to differentembodiments or not, is contemplated to implement and practicecontemplated embodiments. Furthermore, although embodiments disclosedherein may achieve advantages over other possible solutions or over theprior art, whether or not a particular advantage is achieved by a givenembodiment is not limiting of the scope of the present disclosure. Thus,the following aspects, features, embodiments and advantages are merelyillustrative and are not considered elements or limitations of theappended claims except where explicitly recited in a claim(s).

Aspects of the present disclosure may take the form of an entirelyhardware embodiment, an entirely software embodiment (includingfirmware, resident software, micro-code, etc.) or an embodimentcombining software and hardware aspects that may all generally bereferred to herein as a “circuit,” “module,” or “system.” Furthermore,aspects of the present disclosure may take the form of a computerprogram product embodied in one or more computer readable medium(s)having computer readable program code embodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples a computer readable storage medium include: anelectrical connection having one or more wires, a hard disk, a randomaccess memory (RAM), a read-only memory (ROM), an erasable programmableread-only memory (EPROM or Flash memory), an optical fiber, a portablecompact disc read-only memory (CD-ROM), an optical storage device, amagnetic storage device, or any suitable combination of the foregoing.In the current context, a computer readable storage medium may be anytangible medium that can contain, or store a program.

While the foregoing is directed to embodiments of the presentdisclosure, other and further embodiments of the disclosure may bedevised without departing from the basic scope thereof, and the scopethereof is determined by the claims that follow.

What is claimed is:
 1. A system comprising: one or more processors and memory storing one or more instructions that, when executed on the one or more processors, cause the system to: send one or more web pages for display to a user via a network during an interaction session between the user and an application, wherein the one or more web pages include elements for collecting response data from the user; receive, via the web pages, response data from the user, collect, via the application, additional data that characterizes user behavior during the interaction session; generate a composite data set from the response data and the additional data; determine, via a first machine-learning model based on the composite data set, a retention-prediction value indicating a probability that the user will complete a target action in the application before the interaction session terminates; determine that the retention-prediction value satisfies a threshold condition; determine, via a second machine-learning model based on the composite data set and the retention-prediction value determined by the first machine-learning model, a predicted reason the interaction session is likely to terminate before the user completes the target action; determine, via a third machine-learning model based on the composite data set and the predicted reason determined by the second machine-learning model, an intervention action for increasing the probability that the user will complete the target action before the interaction session terminates; and perform, via the application, the intervention action.
 2. The system of claim 1, wherein the instructions, when executed on the one or more processors, further cause the system to: determine, via a fourth machine-learning model based on the composite data set, a next action the user is anticipated to perform in the application; and alter at least one aspect of the one or more web pages to facilitate user performance of the next action.
 3. The system of claim 2, wherein the instructions, when executed on the one or more processors, further cause the system to: fetch, via the application, data related to the next action determined via the fourth machine-learning model.
 4. The system of claim 1, wherein determining the intervention action via the third machine-learning model includes: inputting a set of input features that includes the retention-prediction value into the third machine-learning model.
 5. The system of claim 1, wherein the instructions, when executed on the one or more processors, further cause the system to: receive, via the web pages, updated response data from the user; collect, via the application, updated additional data that characterizes user behavior during the interaction session; generate an updated composite data set that includes both the updated response data and the updated additional data; and determine, via the first machine-learning model based on the updated composite data set, an updated retention-prediction value indicating an updated probability that the user will complete the target action in the application before the interaction session terminates.
 6. The system of claim 5, wherein the instructions, when executed on the one or more processors, further cause the system to: subtract the retention-prediction value from the updated retention-prediction value to determine a difference; divide the difference by a time interval to determine a rate of change; and determine an updated time interval based on the rate of change.
 7. The system of claim 1, wherein performing the intervention action includes: open, via the application, a messaging interface; and establish a network connection with a live support agent to allow the user to communicate with the live support agent through the messaging interface.
 8. A non-transitory computer-readable storage medium containing instructions that, when executed by one or more processors, perform an operation comprising: sending one or more web pages for display to a user via a network during an interaction session between the user and an application, wherein the one or more web pages include elements for collecting response data from the user; receiving, via the web pages, the response data from the user; collecting, via the application, additional data that characterizes user behavior during the interaction session; generating a composite data set from the response data and the additional data; determining, via a first machine-learning model based on the composite data set, a retention-prediction value indicating a probability that the user will complete a target action in the application before the interaction session terminates; determining that the retention-prediction value satisfies a threshold condition; determining, via a second machine-learning model based on the composite data set and the retention-prediction value determined by the first machine-learning model, a predicted reason the interaction session is likely to terminate before the user completes the target action; determining, via a third machine-learning model based on the composite data set and the predicted reason determined by the second machine-learning model, an intervention action for increasing the probability that the user will complete the target action before the interaction session terminates; and performing, via the application, the intervention action.
 9. The non-transitory computer-readable storage medium of claim 8, wherein the operation further comprises: determining, via a fourth machine-learning model based on the composite data set, a next action the user is anticipated to perform in the application; and altering at least one aspect of the one or more web pages to facilitate user performance of the next action.
 10. The non-transitory computer-readable storage medium of claim 9, wherein the operation further comprises: fetching, data related to the next action determined via the fourth machine-learning model.
 11. The non-transitory computer-readable storage medium of claim 8, wherein determining the intervention action via the third machine-learning model includes: inputting a set of input features that includes the retention-prediction value into the third machine-learning model.
 12. The non-transitory computer-readable storage medium of claim 8, wherein the operation further comprises: receiving, via the web pages, updated response data from the user; collecting, via the application, updated additional data that characterizes user behavior during the interaction session; generating an updated composite data set that includes both the updated response data and the updated additional data; and determining, via the first machine-learning model based on the updated composite data set, an updated retention-prediction value indicating an updated probability that the user will complete the target action in the application before the interaction session terminates.
 13. The non-transitory computer-readable storage medium of claim 12, wherein the operation further comprises: subtracting the retention-prediction value from the updated retention-prediction value to determine a difference; dividing the difference by a time interval to determine a rate of change; and determining an updated time interval based on the rate of change.
 14. The non-transitory computer-readable storage medium of claim 8, wherein performing the intervention action includes: opening, via the application, a messaging interface; and establishing a network connection with a live support agent to allow the user to communicate with the live support agent through the messaging interface.
 15. A method comprising: sending one or more web pages for display to a user via a network during an interaction session between the user and an application, wherein the one or more web pages include elements for collecting response data from the user; receiving, via the web pages, the response data from the user; collecting, via the application, additional data that characterizes user behavior during the interaction session; generating a composite data set from the response data and the additional data; determining, via a first machine-learning model based on the composite data set, a retention-prediction value indicating a probability that the user will complete a target action in the application before the interaction session terminates; determining that the retention-prediction value satisfies a threshold condition; determining, via a second machine-learning model based on the composite data set and the retention-prediction value determined by the first machine-learning model, a predicted reason the interaction session is likely to terminate before the user completes the target action; determining, via a third machine-learning model based on the composite data set and the predicted reason determined by the second machine-learning model, an intervention action for increasing the probability that the user will complete the target action before the interaction session terminates; and performing, via the application, the intervention action.
 16. The method of claim 15, further comprising: determining, via a fourth machine-learning model based on the composite data set, a next action the user is anticipated to perform in the application; and altering at least one aspect of the one or more web pages to facilitate user performance of the next action.
 17. The method of claim 16, further comprising: fetching, data related to the next action determined via the fourth machine-learning model.
 18. The method of claim 15, wherein determining the intervention action via the third machine-learning model includes: inputting a set of input features that includes the retention-prediction value into the third machine-learning model.
 19. The method of claim 15, further comprising: receiving, via the web pages, updated response data from the user; collecting, via the application, updated additional data that characterizes user behavior during the interaction session; generating an updated composite data set that includes both the updated response data and the updated additional data; and determining, via the first machine-learning model based on the updated composite data set, an updated retention-prediction value indicating an updated probability that the user will complete the target action in the application before the interaction session terminates.
 20. The method of claim 19, further comprising: subtracting the retention-prediction value from the updated retention-prediction value to determine a difference; dividing the difference by a time interval to determine a rate of change; and determining an updated time interval based on the rate of change. 